[blog] Introducing inf2 runtime blog post
#540
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
spillai
added
docs
spillai
force-pushed
the
spillai/0.2.0-dev-inf2-blog
branch
from
094fac5
to
05d82b7
Compare
outtanames
approved these changes
Feb 1, 2024
|
|
||
| [AWS Inferentia2](https://aws.amazon.com/en/ec2/instance-types/inf2/) (Inf2 for short) is the second-generation inference accelerator from AWS. Inf2 instances raise the performance of Inf1 (originally launched in 2019) by delivering 3x higher compute performance, 4x larger total accelerator memory, up to 4x higher throughput, and up to 10x lower latency. Inf2 instances are the first inference-optimized instances in Amazon EC2 to support scale-out distributed inference with ultra-high-speed connectivity between accelerators. | ||
|
|
||
| Relative to the [AWS G5 instances](https://aws.amazon.com/ec2/instance-types/g5/) ([NVIDIA A10G](https://www.nvidia.com/en-us/data-center/products/a10-gpu/)), Inf2 instances promise up to 50% better performance-per-watt. Inf2 instances are ideal for applications such as natural language processing, recommender systems, image classification and recognition, speech recognition, and language translation that can take advantage of scale-out distributed inference. |
There was a problem hiding this comment.
should we quote the inf2 numbers here against say an A100 for reference?
There was a problem hiding this comment.
Will do this once we have the profiling numbers to compare, I could only find this stat relative to G5.
|
|
||
| ## 📦 Deploying a model on Inferentia2 with NOS | ||
|
|
||
| Deploying models on AWS Inferentia2 chips presents a unique set of challenges, distinctly different from the experience with NVIDIA GPUs. This is primarily due to the lack of a mature toolchain for compiling, profiling, and deploying models onto these specialized ASICs. To effectively utilize the AWS Inferentia2 chips, custom model tracing and compilation are essential steps. This process demands a deep understanding of the deployment toolchain, including PyTorch IR op-support and the [AWS Neuron SDK](https://github.com/aws-neuron/aws-neuron-sdk), to optimize model performance fully. NOS aims to bridge this gap and streamline the deployment process, making it more accessible for developers to leverage the powerful inference capabilities of AWS Inferentia2 for their inference workloads and expose easy-to-use gRPC/RESTful services in a straightforward manner. |
There was a problem hiding this comment.
[nit] distinctly different -> distinct. Also wouldn't say 'lack of mature toolchain', maybe just point out that the Neuron SDK is very differnt from the torch cuda ecosystem.
| | Model | Cloud Instance | Spot | Cost / hr | Cost / month | # of Req. / $ | | ||
| | ----- | -------------- | ---- | --------- | ------------ | ---------- | | ||
| | [BAAI/bge-small-en-v1.5](https://huggingface.co/BAAI/bge-small-en-v1.5) | `inf2.xlarge` | - | $0.75 | ~$540 | ~685K / $1 | | ||
| | **[BAAI/bge-small-en-v1.5](https://huggingface.co/BAAI/bge-small-en-v1.5)** | `inf2.xlarge` | ✅ | **$0.32** | **~$230** | ~1.6M / $1 | |
spillai
force-pushed
the
spillai/0.2.0-dev-inf2-blog
branch
from
05d82b7
to
c023b46
Compare
spillai
force-pushed
the
spillai/0.2.0-dev-inf2-blog
branch
from
c023b46
to
3250c5f
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Related issues
Checks
make lint: I've runmake lintto lint the changes in this PR.make test: I've made sure the tests (make test-cpuormake test) are passing.